Examiner column for June 23.
Imagine 1100 men and women in a cavernous convention center,
laboring 8-5, seven days straight with no days off. Does that sound like a
penal labor camp, or an enterprise beloved by teachers, who return year after
year?
Scoring Advanced Placement tests is the yearly activity that only a participant could love—and love it we do. Don’t try to explain why to others, because they will simply not understand.
Part of the appeal is our entry into a zone where there are
no days of the week—just days of the reading. You arrive not on a Friday, but
just before Day 1. Is it Saturday? Sunday? No one keeps track, except to
announce, “Day 4! Halfway there!”
We
score packs of 25 pink booklets in yellow folders, and know we’ve read 50 or
250, we’re not quite sure. As leader of one of the questions, I see statistics
that fill the gaps in our collective memories. Day 4 of this year’s AP
Literature reading, for instance, saw 1077 readers completing 187,676 essays,
on that day alone. By the end of Day 7, we had polished off nearly 1,035, 000
essays written by 345 thousand students.
With
so many readers, and over a million essays, how can College Board guarantee
that each one receives the score it deserves? The answer is all in the samples.
Five days before the reading starts, several of us choose essay samples of each
score point. We compose accompanying explanations, based on the scoring guide,
and spend a day with the 42 table leaders on each of the three questions. Those
table leaders, in turn, spend Day 1 training their 7 or 8 readers to apply both
the scoring guide and sample essays.
The
scoring is consistent year to year, table to table, reader to reader. 45% of
the overall score is based on a difficult multiple-choice exam, further
stabilizing the 1-5 AP score. Statistics show multiple choice/free response
(essay) correlations to table leaders, alerting them to the rare, inconsistent
reader.
What
is the multiple choice/free response (MC/FR) correlation? If the correlation
were 1:1, there would be no need for essays, and AP scores would be based on
multiple- choice alone. The correlation measures how high or low scores on the
standardized part of the test relate to the essay score. A low correlation makes
no sense; few students do very well on one part and poorly on the other.
Generally,
table leaders are happy if the MC/FR correlation is 50%: half the time the
student scoring better on MC, and the other half better on FR. It’s
complicated—but the calculations give credibility to the enterprise. No college
would give credit for scores deemed unreliable.
The
AP test remains the only standardized test I’ve ever trusted—both for the
intelligent way it’s composed, and the way it’s scored. Somewhere along the
line, magic occurs, as well, as hundreds of AP readers spend a week in minimal
comfort to give student essays the fairest reading possible. Their
conscientiousness and care guarantees the program’s longevity—so we can return
next year, and feel the magic once again.
Just thinking about Louisville last week about this time. . . still recovering here. It was good to work with you again.
Posted by: David | June 24, 2010 at 08:25 AM
I still haven't fully unpacked. (This is the longest "reading hangover" yet...) On the other hand, I'm already looking forward to the next "punishment." See you in 11 months!
Posted by: Erica | June 24, 2010 at 09:33 AM